<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
                "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
  <title>Description of cross_validation</title>
  <meta name="keywords" content="cross_validation">
  <meta name="description" content="Cross-validation classification">
  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
  <meta name="generator" content="m2html &copy; 2003 Guillaume Flandin">
  <meta name="robots" content="index, follow">
  <link type="text/css" rel="stylesheet" href="../m2html.css">
</head>
<body>
<a name="_top"></a>
<!-- menu.html . -->
<h1>cross_validation
</h1>

<h2><a name="_name"></a>PURPOSE <a href="#_top"><img alt="^" border="0" src="../up.png"></a></h2>
<div class="box"><strong>Cross-validation classification</strong></div>

<h2><a name="_synopsis"></a>SYNOPSIS <a href="#_top"><img alt="^" border="0" src="../up.png"></a></h2>
<div class="box"><strong>function [subj results] = cross_validation(subj,patname,regsname,selgroup,maskgroup,class_args,varargin) </strong></div>

<h2><a name="_description"></a>DESCRIPTION <a href="#_top"><img alt="^" border="0" src="../up.png"></a></h2>
<div class="fragment"><pre class="comment"> Cross-validation classification

 [SUBJ RESULTS] = CROSS_VALIDATION(SUBJ,PATNAME,REGSNAME,SELGROUP,CLASS_ARGS,MASKGROUP,...)

 Calls the classifier multiple times, training and testing on
 different subsets of the data each time

 Adds the following objects:
 - results structure

 PATNAME is the data that the classifier will be fed as its input

 REGSNAME contains the targets that the classifier will be taught
 and then tested with. It will create one output unit per row. For pure
 classification, these should be binary, one condition per
 timepoint, but there is no error-checking or removal of rest
 built-in here deliberately.

 SELGROUP is the group of selectors that determine which are testing
 and which are training TRs for a given iteration. One selector
 per iteration. 1s = training, 2s = testing. TRs labelled with 0s
 and other values will be excluded from the classification
 Think of each set of selectors in the cell array as a kind of
 temporal mask. Each set of selectors should have the same number
 of TRs as the patterns and regressors, with 1s for the training
 TRs, 2s for the testing TRs and 0s for the TRs that you want to
 ignore. CREATE_XVALID_INDICES will create such a group.

 MASKGROUP is the group of masks, one per iteration, that will be
 used to decide which features are fed to the classifier. For
 instance, NOPEEKING_MULTI_ANOVA generates such a set of masks.

 Note: if you ran a peeking anova, you'll only have one mask, rather
 than a group. If this can't find any members of a group called
 MASKGROUP, it will treat MASKGROUP as the name of an object, and
 look instead for a single mask called MASKGROUP.

 PERFMET_FUNCTS(optional,default = {'perfmet_maxclass'}). The names of the
 performance metric(s) you want to use to decide how well your
 classifier did. Feed in as a cell array of strings containing
 function names that get turned into function handles later

 PERFMET_ARGS(optional,default = {[]}. Accompanying argument-structures, one
 foir each optional perfmet_funct

 See the manual for more documentation about the results
 structure.</pre></div>

<!-- crossreference -->
<h2><a name="_cross"></a>CROSS-REFERENCE INFORMATION <a href="#_top"><img alt="^" border="0" src="../up.png"></a></h2>
This function calls:
<ul style="list-style-image:url(../matlabicon.gif)">
<li><a href="add_results_history.html" class="code" title="function [results] = add_results_history(results,hist_str,displayme)">add_results_history</a>	Adds a line to the results.header.history field</li><li><a href="datetime.html" class="code" title="function [dt] = datetime(seconds)">datetime</a>	Spits out the date and time in yymmdd_HHMM format</li><li><a href="find_group.html" class="code" title="function [matches] = find_group(subj,objtype,groupname)">find_group</a>	Returns a list of names of objects from this group</li><li><a href="get_masked_pattern.html" class="code" title="function [masked_pat] = get_masked_pattern(subj,patname,maskname)">get_masked_pattern</a>	Returns the pattern's voxels allowed by the mask</li><li><a href="get_mat.html" class="code" title="function [mat] = get_mat(subj,objtype,objname)">get_mat</a>	Returns the MAT field of the object</li><li><a href="get_number.html" class="code" title="function [objno] = get_number(subj,objtype,objname,varargin)">get_number</a>	Internal function - find the cell index of an object</li><li><a href="propval.html" class="code" title="function [combined_struct user_struct undefaulted_struct] = propval(user_propvals,defaults_struct)">propval</a>	This deals with property/value pairs of optional arguments.</li></ul>
This function is called by:
<ul style="list-style-image:url(../matlabicon.gif)">
<li><a href="tutorial_easy.html" class="code" title="function [subj results] = tutorial_easy()">tutorial_easy</a>	Tutorial script to accompany TUTORIAL_EASY.HTM</li></ul>
<!-- crossreference -->

<h2><a name="_subfunctions"></a>SUBFUNCTIONS <a href="#_top"><img alt="^" border="0" src="../up.png"></a></h2>
<ul style="list-style-image:url(../matlabicon.gif)">
<li><a href="#_sub1" class="code">function [] = sanity_check(class_args)</a></li></ul>
<h2><a name="_source"></a>SOURCE CODE <a href="#_top"><img alt="^" border="0" src="../up.png"></a></h2>
<div class="fragment"><pre>0001 <a name="_sub0" href="#_subfunctions" class="code">function [subj results] = cross_validation(subj,patname,regsname,selgroup,maskgroup,class_args,varargin)</a>
0002 
0003 <span class="comment">% Cross-validation classification</span>
0004 <span class="comment">%</span>
0005 <span class="comment">% [SUBJ RESULTS] = CROSS_VALIDATION(SUBJ,PATNAME,REGSNAME,SELGROUP,CLASS_ARGS,MASKGROUP,...)</span>
0006 <span class="comment">%</span>
0007 <span class="comment">% Calls the classifier multiple times, training and testing on</span>
0008 <span class="comment">% different subsets of the data each time</span>
0009 <span class="comment">%</span>
0010 <span class="comment">% Adds the following objects:</span>
0011 <span class="comment">% - results structure</span>
0012 <span class="comment">%</span>
0013 <span class="comment">% PATNAME is the data that the classifier will be fed as its input</span>
0014 <span class="comment">%</span>
0015 <span class="comment">% REGSNAME contains the targets that the classifier will be taught</span>
0016 <span class="comment">% and then tested with. It will create one output unit per row. For pure</span>
0017 <span class="comment">% classification, these should be binary, one condition per</span>
0018 <span class="comment">% timepoint, but there is no error-checking or removal of rest</span>
0019 <span class="comment">% built-in here deliberately.</span>
0020 <span class="comment">%</span>
0021 <span class="comment">% SELGROUP is the group of selectors that determine which are testing</span>
0022 <span class="comment">% and which are training TRs for a given iteration. One selector</span>
0023 <span class="comment">% per iteration. 1s = training, 2s = testing. TRs labelled with 0s</span>
0024 <span class="comment">% and other values will be excluded from the classification</span>
0025 <span class="comment">% Think of each set of selectors in the cell array as a kind of</span>
0026 <span class="comment">% temporal mask. Each set of selectors should have the same number</span>
0027 <span class="comment">% of TRs as the patterns and regressors, with 1s for the training</span>
0028 <span class="comment">% TRs, 2s for the testing TRs and 0s for the TRs that you want to</span>
0029 <span class="comment">% ignore. CREATE_XVALID_INDICES will create such a group.</span>
0030 <span class="comment">%</span>
0031 <span class="comment">% MASKGROUP is the group of masks, one per iteration, that will be</span>
0032 <span class="comment">% used to decide which features are fed to the classifier. For</span>
0033 <span class="comment">% instance, NOPEEKING_MULTI_ANOVA generates such a set of masks.</span>
0034 <span class="comment">%</span>
0035 <span class="comment">% Note: if you ran a peeking anova, you'll only have one mask, rather</span>
0036 <span class="comment">% than a group. If this can't find any members of a group called</span>
0037 <span class="comment">% MASKGROUP, it will treat MASKGROUP as the name of an object, and</span>
0038 <span class="comment">% look instead for a single mask called MASKGROUP.</span>
0039 <span class="comment">%</span>
0040 <span class="comment">% PERFMET_FUNCTS(optional,default = {'perfmet_maxclass'}). The names of the</span>
0041 <span class="comment">% performance metric(s) you want to use to decide how well your</span>
0042 <span class="comment">% classifier did. Feed in as a cell array of strings containing</span>
0043 <span class="comment">% function names that get turned into function handles later</span>
0044 <span class="comment">%</span>
0045 <span class="comment">% PERFMET_ARGS(optional,default = {[]}. Accompanying argument-structures, one</span>
0046 <span class="comment">% foir each optional perfmet_funct</span>
0047 <span class="comment">%</span>
0048 <span class="comment">% See the manual for more documentation about the results</span>
0049 <span class="comment">% structure.</span>
0050 
0051 <span class="comment">% This is part of the Princeton MVPA toolbox, released under the</span>
0052 <span class="comment">% GPL. See http://www.csbmb.princeton.edu/mvpa for more</span>
0053 <span class="comment">% information.</span>
0054 
0055 
0056 defaults.perfmet_functs = {<span class="string">'perfmet_maxclass'</span>};
0057 defaults.perfmet_args = {[]};
0058 args = <a href="propval.html" class="code" title="function [combined_struct user_struct undefaulted_struct] = propval(user_propvals,defaults_struct)">propval</a>(varargin,defaults);
0059 
0060 <span class="comment">% Load the Pattern and the Regressors</span>
0061 patterns   = <a href="get_mat.html" class="code" title="function [mat] = get_mat(subj,objtype,objname)">get_mat</a>(subj,<span class="string">'pattern'</span>,patname);
0062 regressors = <a href="get_mat.html" class="code" title="function [mat] = get_mat(subj,objtype,objname)">get_mat</a>(subj,<span class="string">'regressors'</span>,regsname);
0063 
0064 <span class="comment">% Get the names of the selectors</span>
0065 selnames = <a href="find_group.html" class="code" title="function [matches] = find_group(subj,objtype,groupname)">find_group</a>(subj,<span class="string">'selector'</span>,selgroup);
0066 nIterations = length(selnames);
0067 
0068 masknames = <a href="find_group.html" class="code" title="function [matches] = find_group(subj,objtype,groupname)">find_group</a>(subj,<span class="string">'mask'</span>,maskgroup);
0069 <span class="comment">% If there's only one mask and it doesn't belong to a group, use it each time</span>
0070 <span class="keyword">if</span> isempty(masknames)
0071   <span class="keyword">if</span> ~isempty(<a href="get_number.html" class="code" title="function [objno] = get_number(subj,objtype,objname,varargin)">get_number</a>(subj,<span class="string">'mask'</span>,maskgroup))
0072     masknames = cellstr(repmat(maskgroup,[nIterations 1]));
0073     disp( sprintf(<span class="string">'Using the %s mask each time'</span>,maskgroup) );
0074   <span class="keyword">else</span>
0075     error(<span class="string">'MASKGROUP is neither a mask nor a group.  Discuss...'</span>);
0076   <span class="keyword">end</span>
0077 <span class="keyword">end</span>
0078 
0079 <span class="keyword">if</span> length(masknames) ~= length(selnames)
0080   error(<span class="string">'Your selector and mask groups have different numbers of items in them'</span>);
0081 <span class="keyword">end</span>
0082 
0083 <span class="comment">% Initialize the results structure</span>
0084 results.header.experiment = subj.header.experiment;
0085 results.header.subj_id    = subj.header.id;
0086 
0087 <span class="comment">% Just in case the user only has one perfmet and fed it in as a</span>
0088 <span class="comment">% string rather than cell array</span>
0089 <span class="keyword">if</span> ~iscell(args.perfmet_functs) &amp; ischar(args.perfmet_functs)
0090   warning(<span class="string">'Perfmet_functs should be a cell array, not a string - fixing'</span>);
0091   args.perfmet_functs = {args.perfmet_functs};
0092 <span class="keyword">end</span>
0093 
0094 nPerfs = length(args.perfmet_functs);
0095 
0096 <a href="#_sub1" class="code" title="subfunction [] = sanity_check(class_args)">sanity_check</a>(class_args);
0097 
0098 <span class="comment">% Initialize subtotal_perfs - this is going to keep a running tally</span>
0099 <span class="comment">% of the performance summed over iterations, separately for each</span>
0100 <span class="comment">% performance metric</span>
0101 subtotal_perfs = zeros([nPerfs 1]);
0102 
0103 disp( sprintf(<span class="string">'Starting %i cross-validation classification iterations - %s'</span>, <span class="keyword">...</span>
0104          nIterations,class_args.train_funct_name) );
0105 
0106 <span class="keyword">for</span> n=1:nIterations
0107 
0108   fprintf(<span class="string">'\t%i'</span>,n);  
0109   cur_iteration = [];
0110 
0111   <span class="comment">% Set the current selector up</span>
0112   cur_selsname = selnames{n};
0113   selectors = <a href="get_mat.html" class="code" title="function [mat] = get_mat(subj,objtype,objname)">get_mat</a>(subj,<span class="string">'selector'</span>,cur_selsname);
0114 
0115   <span class="comment">% Extract the training and testing indices from the selector</span>
0116   train_idx = find(selectors==1);
0117   test_idx  = find(selectors==2);
0118   rest_idx  = find(selectors==0);
0119   unknown_idx = selectors;
0120   unknown_idx([train_idx test_idx rest_idx]) = [];
0121   <span class="keyword">if</span> length(unknown_idx)
0122     warning( sprintf(<span class="string">'There are unknown selector labels in %s'</span>,cur_selsname) );
0123   <span class="keyword">end</span>
0124 
0125   <span class="comment">% Set the current mask up</span>
0126   cur_maskname = masknames{n};
0127   masked_pats = <a href="get_masked_pattern.html" class="code" title="function [masked_pat] = get_masked_pattern(subj,patname,maskname)">get_masked_pattern</a>(subj,patname,cur_maskname);
0128   
0129   <span class="comment">% Create the training patterns and targets</span>
0130   trainpats  = masked_pats(:,train_idx);
0131   traintargs = regressors( :,train_idx);
0132   testpats   = masked_pats(:,test_idx);
0133   testtargs  = regressors( :,test_idx);
0134 
0135   <span class="comment">% Create a function handle for the classifier training function</span>
0136   train_funct_hand = str2func(class_args.train_funct_name);
0137 
0138   <span class="comment">% Call whichever training function</span>
0139   scratchpad = train_funct_hand(trainpats,traintargs,class_args);
0140 
0141   <span class="comment">% Create a function handle for the classifier testing function</span>
0142   test_funct_hand = str2func(class_args.test_funct_name);
0143   
0144   <span class="comment">% Call whichever training function</span>
0145   [acts scratchpad] = test_funct_hand(testpats,testtargs,scratchpad);  
0146   
0147   <span class="comment">% Run all the perfmet functions on the classifier outputs and store</span>
0148   <span class="comment">% the resulting perfmet structure in a cell</span>
0149   <span class="keyword">for</span> p=1:nPerfs
0150     <span class="comment">% Get the name of the perfmet function</span>
0151     cur_pm_name = args.perfmet_functs{p};
0152     <span class="comment">% Create a function handle to it</span>
0153     cur_pm_fh = str2func(cur_pm_name);
0154     <span class="comment">% Run the perfmet function and get an object back</span>
0155     cur_pm = cur_pm_fh(acts,testtargs,args.perfmet_args{p});
0156     <span class="comment">% Add the function's name to the object</span>
0157     cur_pm.function_name = cur_pm_name;
0158     <span class="comment">% Append this perfmet object to the array of perfmet objects,</span>
0159     <span class="comment">% only using a cell array if necessary</span>
0160     <span class="keyword">if</span> nPerfs==1
0161       cur_iteration.perfmet = cur_pm;
0162     <span class="keyword">else</span>
0163       cur_iteration.perfmet{p} = cur_pm;
0164     <span class="keyword">end</span>
0165     <span class="comment">% Add this iteration's performance to the tally</span>
0166     cur_iteration.perf(p) = cur_pm.perf;
0167     subtotal_perfs(p) = subtotal_perfs(p) + cur_iteration.perf(p);
0168   <span class="keyword">end</span>
0169   
0170   <span class="comment">% Display the performance for this iteration</span>
0171   disp( sprintf(<span class="string">'\t%.2f'</span>,cur_iteration.perf(p)) );
0172 
0173   <span class="comment">% Book-keep the bountiful insight from this iteration</span>
0174   cur_iteration.created.datetime  = <a href="datetime.html" class="code" title="function [dt] = datetime(seconds)">datetime</a>(true);
0175   cur_iteration.train_idx         = train_idx;
0176   cur_iteration.test_idx          = test_idx;
0177   cur_iteration.rest_idx          = rest_idx;
0178   cur_iteration.unknown_idx       = unknown_idx;
0179   cur_iteration.acts              = acts;
0180   cur_iteration.scratchpad        = scratchpad;
0181   cur_iteration.header.history    = []; <span class="comment">% should fill this in xxx</span>
0182   cur_iteration.created.function  = <span class="string">'cross_validation'</span>;
0183   cur_iteration.created.patname   = patname;
0184   cur_iteration.created.regsname  = regsname;
0185   cur_iteration.created.maskname  = masknames{n};
0186   cur_iteration.created.selname   = cur_selsname;
0187   cur_iteration.train_funct_name  = class_args.train_funct_name;
0188   cur_iteration.test_funct_name   = class_args.test_funct_name;
0189   results.iterations(n) = cur_iteration;
0190   
0191 <span class="keyword">end</span> <span class="comment">% for n nIterations</span>
0192 
0193 disp(<span class="string">' '</span>);
0194 
0195 <span class="comment">% Show me the money</span>
0196 results.total_perf = subtotal_perfs / nIterations;
0197 
0198 mainhist = sprintf( <span class="keyword">...</span>
0199     <span class="string">'Cross-validation using %s and %s - got total_perfs - %s'</span>, <span class="keyword">...</span>
0200     class_args.train_funct_name,class_args.test_funct_name, <span class="keyword">...</span>
0201     num2str(results.total_perf'));
0202 
0203 results = <a href="add_results_history.html" class="code" title="function [results] = add_results_history(results,hist_str,displayme)">add_results_history</a>(results,mainhist,true);
0204 
0205 
0206 
0207 <span class="comment">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%</span>
0208 <a name="_sub1" href="#_subfunctions" class="code">function [] = sanity_check(class_args)</a>
0209 
0210 <span class="keyword">if</span> ~isstruct(class_args)
0211   error(<span class="string">'Class_args should be a struct'</span>);
0212 <span class="keyword">end</span>
0213 
0214 <span class="keyword">if</span> ~isfield(class_args,<span class="string">'test_funct_name'</span>) | ~isfield(class_args,<span class="string">'train_funct_name'</span>)
0215   error(<span class="string">'Need to supply training and testing function names'</span>);
0216 <span class="keyword">end</span>
0217 
0218 <span class="keyword">if</span> ~ischar(class_args.test_funct_name) | ~ischar(class_args.train_funct_name)
0219   error(<span class="string">'Training or testing function names have to be strings'</span>);
0220 <span class="keyword">end</span>
0221</pre></div>
<hr><address>Generated on Tue 06-Sep-2005 22:11:31 by <strong><a href="http://www.artefact.tk/software/matlab/m2html/" target="_parent">m2html</a></strong> &copy; 2003</address>
</body>
</html>